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ABSTRACT 
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English-as-a-Second-Languages (ESL) Tutor, an ILTS for ESL. The Student Model 
is based on student subject matter performance and provides feedback and 
remedial exercises suited to learner expertise. The paper further reports on 
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1. Introduction 

Individualized language instruction has long been recognized as a significant advantage of Computer-Assisted 
Language Learning (CALL) over more traditional workbook tasks. A "one size fits all" approach is not 
appropriate for a learning environment. Students learn at their own pace and often, work for their own purposes. 
Learners also vary with respect to prior language experience, aptitude, and/or learning styles and strategies. 
According to the Individual Differences Theory as described by Oxford (1995), if learners learn differently, 
then they likely benefit from individualized instruction. 

Despite the need for an individualized learning environment. Student Modeling has not been a strong 
focus of CALL. One likely reason is that in order for a computer program to adapt itself to different learner 
needs, the system needs a dynamic model of the strengths and weaknesses of the learner (McCalla & Greer, 
1992). Even when it comes to Intelligent Language Tutoring Systems (ILTSs), only a few have employed 
Student Models to individualize the learning environment. According to Holland & Kaplan (1995), this is likely 
due to the challenging task of representing the domain knowledge itself, the module which contains facts and 
information about the language being taught. If the grammar is not accurate and complete, even a precise 
Student Model cannot compensate. For instance, Holland (1994) states that a system which does not detect 
ambiguous errors accurately will obscure a Student Model. 

There are a number of modeling techniques that can be implemented in a computer program. The 
system can model subject matter performance, students' learning strategies and/or cognitive styles. ILTSs 
primarily model subject matter performance, that is, students’ surface errors. While such a Student Model might 
not be complete, it assists in individualizing the language learning process and “is sufficient to model the 
student to the level of detail necessary for the teaching decisions we are able to make” (Elsom-Cook, 1993: 
238). 

In this paper, we describe the Student Model of the ESL Tutor, our Web-based ILTS for English as a 
Second Language (ESL). The ESL Tutor analyzes sentences from the student and detects grammatical and other 
errors. The feedback modules of the system correlate the detailed output of the linguistic analysis with an error- 
specific feedback message. The Student Model is based on student subject matter performance. It provides 
feedback and remediation suited to learner expertise. 

In the following section, we provide examples of CALL systems that employ Student Models and 
discuss their distinct emphasis. In section 3, we describe the architecture of the Student Model of the ESL Tutor. 
Section 4 reports on a study in which we determined the extent to which our Student Model addresses the need 
for an individualized language learning environment. Concluding comments can be found in section 5. 
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2. Student Modeling and Intelligent Language Tutoring Systems 

In analyzing Student Models, McCalla (1992) makes a distinction between implicit and explicit Student 
Modeling which is particularly useful in classifying the Student Models in ILTSs. 
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An implicit Student Model is static, in the sense that the Student Model is reflected in the design 
decisions inherent to the system and derived from a designer’s point of view. For instance, in an ILTS the native 
language of the learner can be encoded as a bug model that includes frequently made errors and ultimately 
diagnoses them. 

In contrast, an explicit Student Model is dynamic. It is a representation of the learner which is used to 
drive instructional decisions. For ILTSs, for instance, the Student Model can assist in guiding the student 
through remedial exercises or it can adjust instructional feedback suited to the level of the learner. In either 
case, the decisions are based on the previous performance history of the learner. The following discussion will 
provide examples of ILTSs which have implemented implicit and explicit Student Models. 

2.1 Implicit Student Models 

Implicit Student Modeling has been applied to ILTSs to diagnose errors. For example, in Catt & Hirst’s (1990) 
system Scripsi the native language of the student represents the learner model. It is used to model the learner’s 
interlanguage. With regard to Student Modeling, the pitfall of such an implementation is that it is a static 
conception. The system’s view of the learner cannot change across interactions with the system. It has no 
impact on instructional decisions and provides only a gross individualization of the learning process when 
ideally, a Student Model is dynamic (Holt et al., 1994). 

In a more individualized example. Bull (1994) developed a system that teaches clitic pronoun 
placement in European Portuguese. The Student Model is based on the system’s and the student’s belief 
measures, language learning strategies, and language awareness. 

The system’s belief measure is comprised of the proportion of incorrect/correct uses of the rule; the 
students provide the data for the student’s belief measure, being required to state their confidence in their 
answer when entering sentences. Learners also identify their preferred learning strategies when using the 
program. According to Bull (1994), language awareness is achieved by allowing the student access to all 
information held in the system. The information, however, is not used to drive the instructional process. A 
number of studies have also shown that students tend to not take advantage of the option to access additional 
information. For example, Cobb & Stevens (1996) found that in their reading program learners' use of self- 
accessible help was virtually non-existent, in spite of their previously having tried it in a practice session, and 
also having doubled their reading performance as compared to either a no help or dictionary help option in the 
practice session. 

2.2. Explicit Student Models 

In developing an explicit Student Model one typically starts by making some initial assumptions based on 
pretests or stereotypical postulations about the learner. For example, initially every student could be assessed as 
an intermediate. During the instructional process, the Student Model adjusts to student’s behaviour moving to a 
novice or expert profile, as appropriate. This technique is used in explicit Student Models to make instructional 
decisions. 

Explicit Student, Modeling has been used in a number of ILTSs, primarily in the form of tracking. 
Tracking can be as simple as calculating percentages of correct answers or more sophisticatedly, identifying 
particular errors which occurred in the student’s input. The information is then used to alter the instructional 
process, either in the form of further language tasks or feedback. 

Explicit Student Modeling is found in the system The Faulty Article Tutor (Kurup, Greer & McCalla, 
1992) which teaches correct article use in English. The system presents the student with scenarios whereby the 
student must select the correct article form and the appropriate rule. The tutor keeps an error count and selects 
the scenarios on the basis of the performance of the student; thus the path through the program is individualized 
by altering the instructional process according to prior performance of the student. 

Bailin (1988, 1990) in his system Verbcon/Diagnosis also employs the tracking method. Diagnosis 
provides practice in using English verb forms in written texts. All verbs are presented in their infinitival form 
challenging the student to provide the appropriate verb form. The system tracks the most frequent error 
occurrence and the context in which the error occurred. The information is used to provide informative 
feedback based on contrasting correct and ungrammatical uses of tenses. In addition. Diagnosis suggests 
exercises to help with the remediation process. 

In the following section we describe the ESL Tutor and discuss the modeling technique used. 




3 



3. The ESL Tutor 



The goal of the ILTS we have developed for ESL is to provide meaningful and interactive vocabulary and 
grammar practice for second language learners. The ESL Tutor analyzes sentences from the student and detects 
grammatical and other errors. The feedback modules of the system correlate the detailed output of the linguistic 
analysis with an error-specific feedback message. 




Figure 1 : System Feedback for the Intermediate Learner 

In the ESL Tutor^ feedback is individualized through an adaptive, explicit Student Model, which 
monitors a user’s performance over time across different grammatical constructs. This record of strengths and 
weaknesses is used to tailor feedback messages to learner expertise within a framework of guided discovery 
learning: a beginner student will receive the most explicit feedback while the instructional messages for the 
expert will merely hint at the error. The feedback aimed at the beginner will also contain less technical 
terminology than that for the intermediate and expert. For example. Figure (1) shows a feedback message for an 
intermediate student, which indicates incorrect subject-verb agreement. 

In contrast to Figure (1), the feedback message for the beginner learner will provide less linguistic 
terminology and state that The verb IS is not correct here. For the advanced learner, the feedback will provide 
less of a clue and simply display There is an agreement error in this sentence. 

In the following section we discuss the technique employed in the Student Model of the ESL Tutor. 

3.1. The Student Model of the ESL Tutor 

The Student Model of the ESL Tutor dynamically evolves based on the student’s performance. The information 
in the model is used for two main functions: modulation of instructional feedback and assessment and 
remediation. 

The Student Model keeps track of an individual student’s performance on a variety of grammar skills; 
from subject-verb agreement to passives to count/mass nouns. A student has a score for each grammar skill. 
This score ranges from 0 - n, where we have set n to 30. The score increases when the student provides 
evidence of a successful use of that grammar skill, and decreases when the student provides evidence of an 
unsuccessful use of that grammar skill. The amount by which a student's score increases or decreases can vary 
depending on the current value of the score. Initially, we set all scores to an intermediate level. 

For the purposes of modulating instructional feedback, we identity 3 categories of scores. Scores from 
0-10 are assigned to the novice category, 1 1-20 to the intermediate category, and 21-30 to the expert category. 
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When a student makes an error on a particular grammar skill, the message they receive depends on their score 
for that skill. If they are ranked as novice, they will receive a more informative message than if they are ranked 
as an expert. Since the score for each grammar skill is independent of the score for the other grammar skills, a 
student may be expert at subject-verb agreement, but novice at forming the passive - and receive the appropriate 
message. 

The score information is also used for a variety of remediation and assessment tasks. By comparing the 
Student Model at the beginning and end of a session, we can provide a summary of the mistakes that a student 
made during that session. In our current system, these are summarized into general categories such as ’'Verb 
Tenses”, "Pronouns”, etc. These groups are set by means of a parameter file. Similarly, we can also identify the 
grammar skills where the student was correct and provide a "positive" of what the student did right. At present 
we show a list of the errors at the end of each exercise set. 

Further, one can also examine the Student Model overall and identify the current strengths and 
weaknesses of the student. We identify the strengths of a student as the five highest scoring grammar skills that 
have a score greater than 15 (half of the total scale). We identify the weaknesses of a student as the 5 lowest 
scoring grammar skills that have a score less than 15. Students can access this information. 

Finally, the Student Model information can also be used to provide exercises to the student which 
focus on their areas of weaknesses. Instead of repeating the same exercise which the student made the mistake 
on, the ESL Tutor has the capacity to identify examples which require the same grammar skill. This avoids the 
problem of the student rote learning the solution to a particular example, without actually learning the general 
solution. We have not yet implemented this functionality in the ESL Tutor. 



4. Evaluation 

The Student Model of the ESL Tutor is based on our German system (Heift & Nicholson, 2001) which has been 
tested extensively. In one of the studies, we determined the extent to which the Student Model addresses the 
need for an individualized language learning environment. 33 students participated in the study and a total of 
1352 sentences were considered for analysis. 

When analyzing the data with respect to individualized instruction, we were interested in the types of 
errors that occurred during practice and their distribution with respect to the three learner levels: beginner, 
intermediate and advanced. 

The error break-down in Table (1) shows that students were most often at the intermediate level, which 
is not surprising since each student is initially placed at the intermediate level. Nonetheless, approximately one 
third, or 30%, of the time, students either required more elaborate feedback suited to the beginner learner, or, in 
the case of the advanced learner, less detailed feedback was sufficient to correct the errors. Moreover, and 
although not illustrated in Table (1), ten students or 30.3% of all participants received remedial exercises for at 
least one of the six chapters. 





Beginner 


Intermediate 


Advanced 


Total 


% 


Direct Objects (gender, number, case) 


64 


226 


1 


291 


21.5% 


Subject-Verb Agreement (person, number) 


27 


188 


63 


278 


20.6% 


Prepositional Phrases: Dative (gender, number, case) 


48 


185 


1 


234 


17.3% 


Indirect Objects (gender, number, case) 


42 


97 


7 


146 


10.8% 


Subjects (gender, number, case) 


3 


82 


43 


128 


9.5% 


Missing Words 


17 


37 


12 


66 


4.9% 


Prepositional Phrases: Two-way (gender, number, case) 


21 


47 




68 


5.0% 


Prepositional Phrases: Accusative (gender, number, case) 


15 


39 




54 


4.0% 


Extra Words 


11 


19 


11 


41 


3.0% 


Word Order 


10 


16 


10 


36 


2.7% 


Auxiliaries {to have\s. to be) 


1 


6 




7 


0.5% 


Verb complements (infinitive vs. past participle) 


1 


2 




3 


0.2% 




260(19.2%) 


944 (69.8 %) 


148(11%) 


1352 


100% 



Table 1: Break-down of Grammar Errors 
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The data further indicate that most errors occurred with direct objects (21.5%) and subject- verb 
agreement (20.6%). However, these were the most frequent constructions contained in the 120 exercises of this 
study. For instance, only chapters 5 and 6 (40 exercises in total) focus on the present perfect and modals. These 
constructions are not contained in any of the previous chapters, thus there is less opportunity for errors with 
these grammar topics than, for example, subject-verb agreement. 

It is interesting, however, to consider the number of grammar errors made by each learner level. From 
a learning perspective, the data in Table (1) indicate three distinct groups: 

1. those grammar topics where the error distribution for the beginner and advanced levels is fairly 
balanced (missing words, extra words, word order, auxiliaries, verb complements), 

2. those grammar points where students are far more often at the beginner than the advanced level 
(direct and indirect objects, accusative, dative, two-way prepositions), and 

3. those grammatical constructions where students are far more often at the advanced than the 
beginner level (subject-verb agreement, subjects). 

The data of our Student Model also allow us to gain insight into students' performance on a particular 
grammar skill over time. For example. Figure (2) illustrates the performance on direct objects by four students 
who were randomly selected from our data set. The x-axis displays the five chapters that contain direct object 
constructions and the y-axis shows the scores which correspond to the three learner levels. The graphs indicate 
that one of the students stayed at the intermediate level throughout practice. In contrast, the remaining three 
students shifted from the intermediate to the advanced level. The data confirm that while there is variation 
across learners each student also changes performance levels as s/he progresses through the course. 




The data support the need for an individualized system which makes subtle distinctions between 
learners and error types. Our Student Model has a number of advantages. It takes into account students’ past 
performance, and by adjusting the score value to be incremented or decremented, it is adaptable to a particular 
grammatical constraint in an exercise or the pedagogy of a particular instructor. For example, a language 
instructor might rate some errors more salient than others in a given exercise. In such an instance, the 
increment/decrement of some grammar constraints can be tuned to change their sensitivity. 

The main strength of our Student Model, however, is that a single erroneous message will not 
drastically change the overall assessment of the student. The Student Model indicates precisely which 
grammatical violations occurred, allowing for a fine-grained assessment of student competency. In 
consequence, a student can be at a different level for each given grammar constraint reflecting her performance 
of each particular grammatical skill. This subtlety of evaluation is desirable in a language teaching environment 
because as the student progresses through a language course a single measure is not sufficient to capture the 
knowledge attained and to distinguish among learners. The Student Model aids in directing each student toward 
error-specific and individualized remediation. 
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5. Conclusion 



In this paper we provided examples of Student Models that have been employed in CALL over the past decade. 
We introduced our ESL Tutor, an ILTS that provides error-specific and individualized feedback. The Student 
Model of the ESL Tutor is based on learner performance history and makes system decisions accordingly. 

A study in which we evaluated the extent to which our Student Model addresses the need for an 
individualized language learning environment emphasizes the importance of an adaptive language learning 
system that considers user diversity. Approximately one third, or 30%, of the time, students either required 
more elaborate feedback suited to the beginner learner, or, in the case of the advanced learner, less detailed 
feedback was sufficient to correct the errors. The data further confirm that while there is variation across 
learners individual students also change performance levels as they progress through a course. 

Our study also provided some interesting insights into the error typology of different learner levels. 
Due to the constrained environment of the exercises of our system where students select from a given pool of 
vocabulary and grammatical structures, errors in omission, insertion and word order were less frequent than 
other grammar errors. Fewer errors occurred overall and thus the error distribution with respect to beginner and 
advanced levels was fairly balanced. We are currently establishing a similar error typology for our ESL Tutor. 
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